Method for detecting and separating the shadow of moving objects in a sequence of digital images

ABSTRACT

For detecting and separating the shadow of moving objects in a sequence of digital images, a sequence of background images is first determined from a sequence of images, this sequence of background images containing only the non-moving image background of the sequence of images but not the moving objects. Object edges are then extracted from the sequence of images in that in each case an edge gray-value image is determined for each image of the sequence of images and for each respective corresponding background image of the sequence of background images. Using vertical or horizontal projection of these edge gray-value images, corresponding vertical or horizontal edge histograms are determined in each case for the edge gray-value images. Edges which do not belong to moving objects or shadows of moving objects are eliminated, in that the edge histograms of a background image are subtracted from the corresponding edge histograms of the image, belonging to the same instant, of the sequence of images and in that the differential edge histograms formed in this manner are subjected to a threshold value discrimination, by which means those edges are eliminated which do not correspond to the actual moving objects but rather to a shadow of a moving object.

BACKGROUND OF THE INVENTION

In the processing of sequences of digital images from highway trafficscenes for the purposes of detection, tracking and classification ofvehicles, there is often the disturbing effect that the vehicles throw ashadow on the highway in the event of there being direct sunshine, theshadow not being able to be separated in a simple way from the vehicle.As a result, a classification of vehicles according to their geometricaldimensions, in particular according to their width, is often made moredifficult, if not impossible.

Previous approaches to solving this problem fall back on vehicle models(G. D. Sullivan, K. D. Baker: Model-based vision: using cues to selecthypotheses. SPIE Vol. 654 Automatic Optical Inspection (1986), pp.272-277), which are brought into coincidence with the image (matching),as a result of which separating the shadow is not necessary. Thisapproach has the disadvantage, however, that it is verycomputation-intensive and that, given the present state of developmentof image processing hardware, an implementation of such methods in realtime within a useful cost frame does not appear possible.

SUMMARY OF THE INVENTION

The present invention is therefore based on the object of specifying amethod for detecting and separating the shadow of moving objects in asequence of digital images, which is suitable for application in theabove-mentioned image processing methods and which can be implemented inreal time on the basis of cost-effective hardware modules. According tothe invention, this object is achieved by a method for detecting andseparating the shadow of moving objects in a sequence of digital imageshaving the following steps.

In this method, a sequence of background images is first determined froma sequence of images, the sequence of background images containing onlythe non-moving image background of the sequence of images but not themoving objects. Object edges are then extracted from the sequence ofimages, in that, for each image of the sequence of images and for therespective corresponding background image of the sequence of backgroundimages, in each case an edge gray-value image is determined. By means ofvertical or horizontal projection of the edge gray-value images,corresponding vertical or horizontal edge histograms are determined ineach case for the edge gray-value images. Edges which do not belong tomoving objects or shadows of moving objects are eliminated, in that theedge histograms of a background image are subtracted from thecorresponding edge histograms of the image, belonging to the sameinstant, of the sequence of images and in that the differential edgehistograms formed in this manner are subjected to a threshold valuediscrimination, by which means those edges are eliminated which do notcorrespond to the actual moving objects but rather to a shadow of amoving object.

To carry out this method, an essentially smaller computational cost isnecessary than in the case of matching of object models to imageregions.

In a preferred embodiment of the invention, the edge gray-value imagesare put into binary form before the determination of the edgehistograms. By means of this measure, the computational cost in theprocessing of the edge histograms can be further reduced.

A further reduction of the computational cost becomes possible if,before carrying out the steps of extracting object edgesn1, those imageregions are determined which correspond to moving objects - ifappropriate including their shadow - and in which the remaining stepsare applied only to these image regions.

In a further preferred embodiment of the method according to theinvention, edge histograms which belong to one and the same movingobject are combined into a time sequence of edge histograms and theelimination of shadow edges is improved by means of an identification ofedges corresponding to each other.

In a further preferred embodiment of the method, the elimination of theshadow edges is improved further in that preknowledge about the lateralposition of the shadow edges with reference to the object edges is used.This preknowledge can preferably be learned and continuously checked bydetecting the lateral position of the shadow edges in an adaptive mannerfrom the sequence of images. By means of this measure, the practicalityand robustness of the method according to the invention can be furtherimproved.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the present invention which are believed to be novel,are set forth with particularity in the appended claims. The invention,together with further objects and advantages, may best be understood byreference to the following description taken in conjunction with theaccompanying drawings, in the several Figures of which like referencenumerals identify like elements, and in which:

FIG. 1 shows in schematic form an edge gray-value image of an image fromthe sequence of images, with the associated vertical and horizontal edgehistograms.

FIG. 2 shows an edge gray-value image belonging to a background image,with the associated horizontal and vertical edge histograms.

FIG. 3 shows schematically the relationship between the edge histogramsof the image of the sequence of images and of the background image withthe differential edge histogram and the edge histogram determined bymeans of threshold value discrimination, using the example of verticaledge histograms.

FIG. 4 shows an example of an image of the sequence of images.

FIG. 5 shows an example of a moving object image, in which the movingobjects are visible as black areas.

FIG. 6 shows the marking of moving objects, including their shadow, bymeans of including rectangles within one image of the sequence ofimages.

FIG. 7 shows a background image belonging to the sequence of imagesconsidered.

FIG. 8 shows an image with the assignment of the vertical histograms tothe moving objects.

FIG. 9 shows the marking of moving objects with the exclusion of theirshadow by means of including rectangles in an image of the sequence ofimages.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention will be explained in more detail in the following text, bymeans of preferred exemplary embodiments and with the aid of thefigures.

Shown in FIG. 4 is a typical highway traffic scene at a specificinstant, as it is recorded by a camera which is erected (for example ona bridge above the highway). Because of the sunlight, incident laterallyfrom the left in this case, the moving objects throw a distinctlyrecognizable shadow to their right side onto the highway. The fact thatthis shadow has a disturbing effect on the determination of thegeometrical dimensions, especially of the width of the moving objects,is shown in particular by a glance at FIG. 5, in which the movingobjects, including their shadows, are shown as black markings in theimage plane. FIG. 6 shows the including rectangles, of the movingobjects, resulting from these conditions, including their shadows, whichlead to considerable overestimates of the dimensions of the movingobjects.

From various publications (W. Feiten et al.: A video-based system forextracting traffic flow parameters, Proc. 13th DAGM 1991, Munich, pp.507-514; K. P. Karmann, A. v. Brandt: Moving object segmentation basedon adaptive reference images, Proc. of EUSIPCO 1990, Barcelona), methodsfor determining sequences of background images belonging to sequences ofimages are known. These sequences of background images have the propertythat moving objects which are contained in the original sequence ofimages are not contained in said sequence of background images.

FIG. 7 shows a background image belonging to the sequence of imagesrepresented by FIG. 4. Such a background image is calculated in anadaptive manner from the original sequence of images, as a result ofwhich all slowly varying changes of the original sequence of images arealso contained in the sequence of background images. The calculation ofsuch sequences of background images is known from the literature andthus represents no particular technical problem in connection with thepresent invention. As can be seen from FIGS. 4, 5 and 6, however, notonly are the moving objects eliminated from the sequence of backgroundimages, but also, if present, the shadows belonging to these objects,thrown in this case onto the highway. For this reason, these shadows,and not just the actual moving objects, are also contained in the objectimages generated by means of the formation of differences from thesequence of images and the sequence of background images. In order toseparate said shadows from the actual moving objects, according to theinvention, an edge gray-value image is now calculated for each image ofthe sequence of images and also for each corresponding image of thesequence of background images, using known methods which are describedin the literature. A series of classical methods for the extraction ofedges from sequences of digital images is described in the book by Y.Shirai, Three-Dimensional computer vision, Springer Verlag, 1987,especially on pages 32 ff. The calculation of edge gray-value imagesthus represents no technical problem in connection with the presentinvention.

FIG. 1 shows the typical course of the edges in an image of the sequenceof images from a highway traffic scene. The edges 11, 12 and 13 mark theedges of the highway and the highway center, respectively, the edges 14to 17 mark a moving object on the left highway and the edges 18, 19 and110 mark the edges of the thrown shadow belonging to the object. Theassociated horizontal (HH1) and vertical (HV1) edge histograms are shownbelow and to the left, respectively, beside the schematicallyrepresented edge gray-value image. Edge histograms of this type aredetermined from the edge gray-value image using standard methods, whichare entirely familiar to those skilled in the art in the field of imageprocessing and therefore require no further representation in connectionwith the present invention. These edge histograms are determined bymeans of vertical or horizontal projection of the edge gray-value image,that is to say by summing the gray values along the corresponding rowsand columns of the digital image.

FIG. 2 shows in a schematic way the edge gray-value image of thecorresponding background image, in which only the edges of the highwayedge markings and of the highway center are to be seen, but not theedges of the moving object and its shadow. Correspondingly, theassociated edge histograms contain few lines or even no lines at all.

FIG. 3 then shows, using the example of vertical histograms, how thehistograms shown in FIGS. 1 and 2 are subtracted from each other and aresubjected to a threshold value discrimination, as a result of which thehistograms HV3 and HV4, respectively, shown in FIG. 3, are formed. Thehistogram HV3 contains only the edges of the moving object and itsshadow and the histogram HV4 contains, after the edge of the shadow hasbeen removed by means of threshold value discrimination, exclusively theedges belonging to the moving object.

After this elimination of the shadow edges, the actual moving objectedges can very easily be found out in the edge gray-value images andalso in the original images of the sequence of images, as a result ofwhich a correct determination of the geometric dimensions of the movingobjects is very simply possible.

In a preferred exemplary embodiment of the method according to theinvention, the edge gray-value images are put into binary form beforethe determination of the edge histograms using standard methods forimage processing. It is plausible that the quantity of data to beprocessed can be decisively reduced in this manner and that thecalculation of the histograms from the images is simplified by this. Afurther advantage of putting into binary form is to be seen in apossible reduction of the influence of statistical noise processes,which can possibly be contained in the original image material.

In a further exemplary embodiment of the invention, carrying out steps,after the first step of determining a sequence of background images, isrestricted to those image regions which correspond to moving objects -if appropriate including their shadow. Such image regions are marked inFIG. 5 as black regions. In a variant of the method according to theinvention, instead of the image regions marked black in FIG. 5, theenclosing rectangles of these image regions can also be used.

In a preferred embodiment of the method according to the invention,shadow edges and/or object edges are tracked in time. By doing this, animprovement of the edge classification with a simultaneous reduction ofthe computational cost is possible. For this purpose, time series ofhorizontal or vertical edge histograms are formed, in whichcorresponding edges are identified and their movement tracked in thecourse of time. For this purpose, various methods for tracking objectsin time series of images can be used, as they are known from theliterature (for example W. Feiten et al., 1991).

In order to improve further the efficiency of the method and to reducethe computational cost as far as possible, a further preferred exemplaryembodiment of the method according to the invention uses preknowledgeabout the occurrence and the lateral position of thrown shadows atobject edges. For this purpose, a hierarchical system architecture isused which has typically three hierarchical levels. The highestprocessing level uses knowledge about the calendar date and the time toassess the possibility of the occurrence of thrown shadows. Iftherefore, because of the season or time of day, the occurrence ofthrown shadows is excluded, the method omits a corresponding analysis.Otherwise, a processing level lying below said level analyzes the movingobjects with respect to typical attributes, such as for example abnormalvalues for the object height or object width. In so doing, the expectedshadow direction and lateral position and the probable shadow shape areused for improving the search for shadow edges in the vicinity of adetected object.

A third processing level, lying below these levels, ensures theprocessing of preknowledge which has been acquired by means of theanalysis of images preceding in time, that is to say in an adaptivemanner. Each shadow which has been detected in one of the precedingimages has, with a larger probability, a successor in one of thefollowing images. By means of a corresponding time tracking of shadowedges, the efficiency of the method can be considerably increased.

In the tracking of the objects through the time series of digitalimages, their positions are precomputed for future instants. In thisarrangement, the following object properties, inter alia, are recorded:

instantaneous position

predicted position in a future image

instantaneous speed

predicted speed in a future image

object width and

a plausibility.

For tracking geometric object properties, spatial coordinates arepreferably used (in contrast to image coordinates) in this case. Theobject dimensions are preferably recorded with the aid of circumscribingrectangles. For the time tracking of all attributes and parameters,Kalman filters are preferably used.

A correct separation of the shadows from the corresponding objectsrequires the use of knowledge about the shadow direction, which resultsfrom the calendar date, the time and from geometrical influencingvariables of the highway traffic scene. For instance, if it is knownthat the shadows lie on the right-hand sides of the moving objects, theedge extraction begins on this side of a segment and generates thecorresponding edge histograms.

After the shadows have been separated from the objects, the objectgeometry (for example width, height) can be accurately determined and anobject description can be established. From the determined shadowdirection, the geometrical relationships of the highway traffic scenecan be determined and used to improve following processing steps.

Finally, various parameters of the method and the image-producinghardware can be optimized and adjusted, taking into account the factthat the vehicles are rigid objects.

In this description, the following publications have been cited:

G. D. Sullivan, K. D. Baker: Model-based vision: using cues to selecthypotheses. SPIE Vol. 654 Automatic Optical Inspection (1986), pp.272-277

W. Feiten et al.: A video-based system for extracting traffic flowparameters, Proc. 13th DAGM 1991, Munich, pp. 507-514

K. P. Karmann, A. v. Brandt: Moving object segmentation based onadaptive reference images, Proc. of EUSIPCO 1990, Barcelona

Y. Shirai, Three-Dimensional computer vision, Springer Verlag, 1987

The present description is followed by listings of computer programs orprogram modules which can be used for carrying out the method describedabove on an electronic computing system.

The invention is not limited to the particular details of the methoddepicted and other modifications and applications are contemplated.Certain other changes may be made in the above described method withoutdeparting from the true spirit and scope of the invention hereininvolved. It is intended, therefore, that the subject matter in theabove depiction shall be interpreted as illustrative and not in alimiting sense. ##SPC1##

What is claimed is:
 1. A method for detecting and separating shadow ofmoving objects in a sequence of digital images, comprising the stepsof:a) determining a sequence of background images from the sequence ofdigital images, said sequence of background images containing only thenon-moving image background of the sequence of images but not the movingobjects; b) determining, for extracting object edges, in each case anedge gray-value image for each image of the sequence of digital imagesand for each respective corresponding background image of the sequenceof background images; c) determining, by means of vertical or horizontalprojection of said edge gray-value images, corresponding vertical orhorizontal edge histograms in each case for the edge gray-value images;d) eliminating edges which do not belong to moving objects or shadows,in that edge histograms of a background image are subtracted fromcorresponding edge histograms of an image, at a common instant in time,of the sequence of digital images, thereby forming differential edgehistograms; e) subjecting the differential edge histograms to athreshold value discrimination, to eliminate those edges which do notcorrespond to actual moving objects but which do correspond to a shadowof a moving object.
 2. The method as claimed in claim 1, wherein theedge gray-value images are put into binary form before the determinationof the edge histograms.
 3. The method as claimed in claim 1, wherein,before carrying out steps b) to e) image regions are determined whichcorrespond to moving objects and wherein steps b) to e) are applied onlyto these image regions.
 4. The method as claimed in claim 1, wherein themethod further comprises forming a time sequence of edge histograms fromthe edge histograms which belong to a common moving object, andimproving elimination of shadow edges by identification and timetracking of corresponding edges.
 5. The method as claimed in claim 1,wherein the method further comprises elimination of shadow edges byevaluating a preknowledge about a lateral position of the shadow edgeswith reference to corresponding object edges.
 6. The method as claimedin claim 4, wherein the method further comprises elimination of shadowedges by evaluating a preknowledge about a lateral position of theshadow edges with reference to corresponding object edges, and whereinthe preknowledge about the lateral position of the shadow edges islearned in an adaptive manner using the time sequence and iscontinuously checked.